Formative e-assessment: Practitioner cases

This paper reports on one aspect of the Joint Information Systems Committee (JISC)-funded project 'Scoping a vision of formative e-assessment', namely on cases of formative e-assessment developed iteratively with the UK education practitioner community. The project, which took place from June 2008 – January 2009, aimed to identify current theories and practices relating to formative assessment of learning where technologies play a key role. The project aimed to scope the 'domain' of formative e-assessment, by developing cases of practice and identifying key formative processes within them, which are affected by the use of technologies. From this analysis, patterns were extracted to inform future software design. A discussion of the key issues emerging from the review of the literature on formative e-assessment, a full account of the project methodology – the design pattern methodology – as well as a critical discussion of the findings – namely the patterns and the role of technology – are the focus of a separate paper (see Daly et al (forthcoming). This paper documents how cases of formative e-assessment were developed during the project by a collaborative methodology involving practitioners from a range of post-16 education contexts. The cases were analysed with reference to key theoretical perspectives on formative assessment, particularly the work of Black and Wiliam (2009). In addition, Laurillard's Conversational Framework (2002, 2007) was used to locate practices of formative assessment within a wider concept of learning and teaching involving technologies, although a detailed discussion of the latter is not within the scope of this paper. Introduction and literature review This paper focuses on the cases which were developed as part of the JISC-funded project ‘Scoping a vision of formative e-assessment’ which aimed to identify current theories and practices relating to formative assessment where technologies play a key role. The project aimed to scope the ‘domain’ of formative e-assessment within post-16 contexts including Higher Education (HE), Further Education (FE) and WorkBased Learning (WBL), by developing cases of practice and identifying key formative processes within them which are affected by the use of technologies. A further aim, not explored in this paper, was then to derive patterns by analysing the cases, and capture the essential features of a range of formative e-assessment practices in patterns which would inform future software development. This paper presents two representative cases, and discusses the contribution they make to understanding the features of formative assessment where technologies have a range of roles in learning and teaching. For a comprehensive project report see Pachler et al (2009). Accepted for publication in Computers & Education on 24-­‐09-­‐09 The literature presents a diverse set of perspectives on the nature and value of formative e-assessment. It is sometimes presented principally as a relatively easy way of introducing e-assessment within an institution, arguing that as ‘low stakes’ assessment it presents less of a threat to institutions. Formative assessment is often presented as being simply serial (or repeated) summative assessment, and it is even referred to as ‘practice’ assessment. However, many authors question the value of such an approach and argue for a view of formative assessment that places much more emphasis on the role of feedback in the learning process (e.g. Maughan, Peet, and Willmott 2001). For others, formative e-assessment is seen as having the potential to support significant changes in the way in which learning occurs in HE. Whitelock (2007), for example, argues for formative e-assessment as a means of promoting self-reflection and students taking control of their own learning, a view which echoes the ideas of Nichol (2006), who proposes a series of design principles for using formative eassessment to support learner self regulation. Writing in 1998 and looking to the future of e-assessment, Bennet (1998) describes three generations of e-assessment: First Generation tests using designs based closely on existing paper-based tests, Next-Generation tests which would use new formats including multimedia, constructed response, automatic item generation and automatic scoring, and Generation “R” (Reinvention) tests which would use complex simulations and intelligent tutors. Simulations such as the those now used in medical licensing examinations (see http://www.usmle.org/Orientation/2009/menu.html) enable detailed formative feedback on the execution of complex tasks, whilst intelligent tutoring systems (see for example http://www.carnegielearning.com/ and http://www.aleks.com/) use cognitive science and statistical techniques to generate detailed models of the student’s understanding and use this to provide formative feedback and to modify the instructional sequence. A rather different approach to forming the future of formative assessment is seen in the call by Elliott (2008) for the incorporation of perspectives based on Web 2.0 technologies and approaches, and the blurring of boundaries between formative and summative assessment. In the Roadmap for e-assessment developed for JISC by Whitelock and Brashar (2006) the authors consulted widely with the UK e-assessment community considering the divergent views of the ways forward for e-assessment in HE. They concluded that the future roles of e-assessment in the medium term lie in the areas of on-demand testing, e-portfolios, student modelling, formative assessment supporting learner autonomy, and diagnostic assessment, and they stress the need for improved forms of feedback informed by pedagogical (or rather androgogical) principles. Within this context JISC funded a range of projects (see http://www.jisc.ac.uk/assessment.html) as part of its e-Learning programme in order to examine various aspects of e-assessment. The project described here is one of several that dealt with formative e-assessment. The more advanced techniques belonging to Bennett’s Generation “R”, such as the use of simulations, were addressed in the Review of Advanced e-Assessment Techniques project (Ripley et al 2009). The use of e-portfolios was addressed in a number of projects and the work of these is summarised in the JISC report ‘Effective Practice with e-Portfolios’ (JISC 2008). Therefore, positioning itself with respect to other ongoing work, the project Accepted for publication in Computers & Education on 24-­‐09-­‐09 reported in this present study concentrated on examples of formative e-assessment where the pedagogy was relatively well developed, but excluded both the use of eportfolios and the use of ‘advanced’ techniques. The work of Black and Wiliam (1998) is widely quoted in the literature in developing an argument for the effectiveness and value of formative e-assessment, though the argument is often made without regard for the nature of the feedback and interaction in teaching and learning contexts. Our own interest in formative e-assessment also starts from Black and Wiliam’s (1998) work and in particular their recent theorisation of formative assessment (Black and Wiliam 2009), which proposes five key strategies: 1 engineering effective classroom discussion, questions, and learning tasks that elicit evidence of learning; 2 providing feedback that moves learners forward; 3 clarifying and sharing learning intentions and criteria for success; 4 activating students as owners of their own learning; and 5 activating students as instructional resources for one another. Core to Black and Wiliam’s developing theory of formative assessment is the concept of ‘moments of contingency’ (Leahy et al, 2005) which they argue to be ‘hinge’ points in the teaching and learning process where the flow of instruction will depend on how students respond and cannot be predetermined (Black and Wiliam, 2009). Moments of contingency contain within them the scope for learners’ understanding to be ‘otherwise’. Contingency is constituted by a multiplicity of factors which are present in the learning context, and which may be social, cognitive and psychological. A learner’s response to a potential learning opportunity (provided for example by teacher questioning, stimulus material, automated scoring of performance or peer comment) is part of their unique engagement in the learning process and is autobiographical. Responsiveness (on the part of the learner, teacher and/or peers) is key to contingency, and is necessary to ‘moments’, which have formative effects on learning. The role of technologies in contributing to the potential for responsiveness within moments of contingency became a focus for analysing the cases, and identifying ‘hinge’ points in the learning process. Our perspective on the use of technology to support formative assessmenthas also been strongly influenced by Lauruillard’s Conversational Framework (Laurillard, 2002, 2007). We view learning as ‘conversational’, consisting of a series of iterative cycles of interaction between teacher/learner/peers in a variety of combinations which may make use of technologies to greater or lesser degrees. Drawing on these perspectives, and for the purposes of developing cases of practice for this study, we defined formative e-assessment as the use of ICT to support the iterative process of gathering and analysing information about student learning by teachers as well as learners and of evaluating it in relation to prior achievement and attainment of intended, as well as unintended learning outcomes. We did so also with a view to allowing the teacher or student to adjust the learning trajectory by becoming activated in a variety of ways. Within this definition, technologies of eassessment are not seen as in themselves being inherently either summative or formative, but rather what is of interest is whether e-assessment is being used summatively or formatively. Accepted for publication in Computers & Education on 24-­‐09-­‐09